Self-Driving Car Engineer Nanodegree

Deep Learning

Project: Build a Traffic Sign Recognition Classifier

In this notebook, a template is provided for you to implement your functionality in stages, which is required to successfully complete this project. If additional code is required that cannot be included in the notebook, be sure that the Python code is successfully imported and included in your submission if necessary.

Note: Once you have completed all of the code implementations, you need to finalize your work by exporting the iPython Notebook as an HTML document. Before exporting the notebook to html, all of the code cells need to have been run so that reviewers can see the final implementation and output. You can then export the notebook by using the menu above and navigating to \n", "File -> Download as -> HTML (.html). Include the finished document along with this notebook as your submission.

In addition to implementing code, there is a writeup to complete. The writeup should be completed in a separate file, which can be either a markdown file or a pdf document. There is a write up template that can be used to guide the writing process. Completing the code template and writeup template will cover all of the rubric points for this project.

The rubric contains "Stand Out Suggestions" for enhancing the project beyond the minimum requirements. The stand out suggestions are optional. If you decide to pursue the "stand out suggestions", you can include the code in this Ipython notebook and also discuss the results in the writeup file.

Note: Code and Markdown cells can be executed using the Shift + Enter keyboard shortcut. In addition, Markdown cells can be edited by typically double-clicking the cell to enter edit mode.


Step 0: Load The Data

In [273]:
# Load pickled data
import pickle

# TODO: Fill this in based on where you saved the training and testing data

#Howard come back and remove the full path name

training_file = "/home/hbutler/udacity/traffic_sign_data/train.p"
validation_file="/home/hbutler/udacity/traffic_sign_data/valid.p"
testing_file = "/home/hbutler/udacity/traffic_sign_data/test.p"

with open(training_file, mode='rb') as f:
    train = pickle.load(f)
with open(validation_file, mode='rb') as f:
    valid = pickle.load(f)
with open(testing_file, mode='rb') as f:
    test = pickle.load(f)
    
X_train, y_train = train['features'], train['labels']
X_valid, y_valid = valid['features'], valid['labels']
X_test, y_test = test['features'], test['labels']

Step 1: Dataset Summary & Exploration

The pickled data is a dictionary with 4 key/value pairs:

  • 'features' is a 4D array containing raw pixel data of the traffic sign images, (num examples, width, height, channels).
  • 'labels' is a 1D array containing the label/class id of the traffic sign. The file signnames.csv contains id -> name mappings for each id.
  • 'sizes' is a list containing tuples, (width, height) representing the original width and height the image.
  • 'coords' is a list containing tuples, (x1, y1, x2, y2) representing coordinates of a bounding box around the sign in the image. THESE COORDINATES ASSUME THE ORIGINAL IMAGE. THE PICKLED DATA CONTAINS RESIZED VERSIONS (32 by 32) OF THESE IMAGES

Complete the basic data summary below. Use python, numpy and/or pandas methods to calculate the data summary rather than hard coding the results. For example, the pandas shape method might be useful for calculating some of the summary results.

Provide a Basic Summary of the Data Set Using Python, Numpy and/or Pandas

In [152]:
### Replace each question mark with the appropriate value. 
### Use python, pandas or numpy methods rather than hard coding the results

###Howard come back and take a look at this

import numpy as np

# Number of training examples
n_train = X_train.shape[0]

# Number of validation examples
n_validation = X_valid.shape[0]

# Number of testing examples.
n_test = X_test.shape[0]

# What's the shape of an traffic sign image?
image_shape = X_train.shape[1:]

# How many unique classes/labels there are in the dataset.
n_classes = len(np.unique(y_train))

print("Number of training examples =", n_train)
print("Number of Validation examples =", n_validation)
print("Number of testing examples =", n_test)
print("Image data shape =", image_shape)
print("Number of classes =", n_classes)
Number of training examples = 34799
Number of Validation examples = 4410
Number of testing examples = 12630
Image data shape = (32, 32, 3)
Number of classes = 43

Include an exploratory visualization of the dataset

Visualize the German Traffic Signs Dataset using the pickled file(s). This is open ended, suggestions include: plotting traffic sign images, plotting the count of each sign, etc.

The Matplotlib examples and gallery pages are a great resource for doing visualizations in Python.

NOTE: It's recommended you start with something simple first. If you wish to do more, come back to it after you've completed the rest of the sections. It can be interesting to look at the distribution of classes in the training, validation and test set. Is the distribution the same? Are there more examples of some classes than others?

In [153]:
class Image_House:
    def __init__(self, classification):
        self.classification = classification
        self.images = []
    def add(self, image):
        self.images.append(image)
    def get(self, index):
        #should add some protection here
        return self.images[index]
   
    
    
 
In [154]:
### Data exploration visualization code goes here.
### Feel free to use as many code cells as needed.
import matplotlib.pyplot as plt
import random
import csv
import numpy as np
import matplotlib.image as mpimg
import cv2
# Visualizations will be shown in the notebook.
%matplotlib inline

def read_label_names(filename):
    with open(filename, newline='', encoding="utf8") as file:
        filereader = csv.reader(file)
        names = [x[1] for x in filereader][1:]
        return names  
       
def sort_images(label_names, image_data):
    image_house_list = []
    for label in label_names:
        image_house_list.append(Image_House(label))
    label_list = y_train
    for image in image_house_list:
        for index, label in enumerate(label_list):
            if label_names[label] == image.classification:
                image.add(image_data[index])
    return image_house_list

def display_images(image_list, numb_images, write_class="Stop"):
    random.seed()
    for image_class in image_storage_list:
        plt.figure(figsize=(15,15))
        for image_number in range(numb_images):
            image = random.randint(0, len(image_class.images) -1)
            ax = plt.subplot(10,numb_images,image_number+1)
            if image_number == 0:
                ax.set_title(image_class.classification, fontsize=10)
            ax.axis('off')
            ax.imshow(image_class.images[image])
            #ax.imshow(cv2.cvtColor(image_class.images[image], cv2.COLOR_RGB2GRAY), cmap='gray')
            if image_class.classification == write_class:
                stop_file_name = "examples/" + write_class.lower() + str(image_number) + ".png"
                mpimg.imsave(stop_file_name, image_class.images[image])
        plt.show()


        
label_names = read_label_names('signnames.csv')

image_storage_list = sort_images(label_names, X_train)

display_images(image_storage_list, 10)
In [155]:
plt.hist(y_train, bins = n_classes)
plt.ylabel("Number Images")
plt.xlabel("Classification")
plt.savefig('examples/visualization.png')

Step 2: Design and Test a Model Architecture

Design and implement a deep learning model that learns to recognize traffic signs. Train and test your model on the German Traffic Sign Dataset.

The LeNet-5 implementation shown in the classroom at the end of the CNN lesson is a solid starting point. You'll have to change the number of classes and possibly the preprocessing, but aside from that it's plug and play!

With the LeNet-5 solution from the lecture, you should expect a validation set accuracy of about 0.89. To meet specifications, the validation set accuracy will need to be at least 0.93. It is possible to get an even higher accuracy, but 0.93 is the minimum for a successful project submission.

There are various aspects to consider when thinking about this problem:

  • Neural network architecture (is the network over or underfitting?)
  • Play around preprocessing techniques (normalization, rgb to grayscale, etc)
  • Number of examples per label (some have more than others).
  • Generate fake data.

Here is an example of a published baseline model on this problem. It's not required to be familiar with the approach used in the paper but, it's good practice to try to read papers like these.

Pre-process the Data Set (normalization, grayscale, etc.)

Minimally, the image data should be normalized so that the data has mean zero and equal variance. For image data, (pixel - 128)/ 128 is a quick way to approximately normalize the data and can be used in this project.

Other pre-processing steps are optional. You can try different techniques to see if it improves performance.

Use the code cell (or multiple code cells, if necessary) to implement the first step of your project.

In [274]:
### Preprocess the data here. It is required to normalize the data. Other preprocessing steps could include 
### converting to grayscale, etc.
### Feel free to use as many code cells as needed.
import numpy as np
import cv2

from sklearn.utils import shuffle

X_train, y_train = shuffle(X_train, y_train)


def normalize(image_list):
    normalized_list = []
    max_val = np.max(image_list)
    min_val = np.min(image_list)
    halfway = (max_val - min_val)/2
    for image in image_list:
        normalized_list.append((image - halfway)/(max_val - min_val))
    return np.asarray(normalized_list)

def histogram_equalization(image_list, adddim=True):
    histogram_list = []   
    clahe = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(1,1))
    for image in image_list:
        img = clahe.apply(image)
        if adddim:
            img = img[:,:,np.newaxis]
        histogram_list.append(img)
    return np.asarray(histogram_list)


def greyscale(image_list):
    greyscale_list = []
    for image in image_list:
        img = np.asarray(cv2.cvtColor(image, cv2.COLOR_RGB2GRAY))
        img = img[:,:,np.newaxis]
        greyscale_list.append(img)
        #greyscale_list.append(np.sum(image/3, axis=3, keepdims=True))
    return greyscale_list

def greyscale_single(image):
    img = np.asarray(cv2.cvtColor(image, cv2.COLOR_BGR2GRAY))
    img = img[:,:,np.newaxis]
    return img


def grayscale(image_list):
    return (np.sum(image_list/3, axis=3, keepdims=True))

def display_processed_images_dual(original_data, processed_data, numb_images, write_class="Stop",colormap=None):
    random.seed()
    #for image_class in processed_data:
    for i in range(len(processed_data)):
        plt.figure(figsize=(100,100))
        for image_number in range(numb_images):
            image = random.randint(0, len(processed_data[i].images) -1)
            ax = plt.subplot(10,numb_images,image_number+1)
            if image_number == 0:
                ax.set_title(processed_data[i].classification, fontsize=10)
            ax.axis('off')
            ax.imshow(np.hstack((processed_data[i].images[image].squeeze(), original_data[i].images[image].squeeze())), cmap=colormap)
            #ax.imshow(cv2.cvtColor(image_class.images[image], cv2.COLOR_RGB2GRAY), cmap='gray')
            if processed_data[i].classification == write_class:
                stop_file_name = "examples/" + "processed_" + write_class.lower() + str(image_number) + ".png"
                mpimg.imsave(stop_file_name, processed_data[i].images[image].squeeze())
        plt.show()
        

def display_processed_images(processed_data, numb_images, write_class="Stop", colormap=None):
    random.seed()
    #for image_class in processed_data:
    for i in range(len(processed_data)):
        plt.figure(figsize=(35,35))
        for image_number in range(numb_images):
            image = random.randint(0, len(processed_data[i].images) -1)
            ax = plt.subplot(10,numb_images,image_number+1)
            if image_number == 0:
                ax.set_title(processed_data[i].classification, fontsize=10)
            ax.axis('off')
            ax.imshow(processed_data[i].images[image].squeeze(), cmap=colormap)
            #ax.imshow(cv2.cvtColor(image_class.images[image], cv2.COLOR_RGB2GRAY), cmap='gray')
            if processed_data[i].classification == write_class:
                stop_file_name = "examples/" + "processed_" + write_class.lower() + str(image_number) + ".png"
                mpimg.imsave(stop_file_name, processed_data[i].images[image].squeeze())
        plt.show()


X_train_grey = greyscale(X_train)
X_valid_grey = greyscale(X_valid)
X_test_grey = greyscale(X_test)

X_train_hist = histogram_equalization(X_train_grey)
X_valid_hist = histogram_equalization(X_valid_grey)
X_test_hist = histogram_equalization(X_test_grey)

X_train_processed = normalize(X_train_hist)
X_valid_processed = normalize(X_valid_hist)
X_test_processed = normalize(X_test_hist)



image_storage_list = sort_images(label_names, X_train_processed)
#processed_image_storage_list = sort_images(label_names, X_train_processed)
#display_processed_images_dual(processed_image_storage_list, image_storage_list, 10, colormap='gray')

display_processed_images(image_storage_list, 10, colormap='gray')

X_train = X_train_processed
X_valid = X_valid_processed
X_test = X_test_processed

Setup Tensor Flow

In [161]:
import tensorflow as tf

EPOCHS = 100
BATCH_SIZE = 128

Model Architecture

In [93]:
from tensorflow.contrib.layers import flatten

def LeNet(x):    
    # Arguments used for tf.truncated_normal, randomly defines variables for the weights and biases for each layer
    mu = 0
    sigma = 0.1
    
    # SOLUTION: Layer 1: Convolutional. Input = 32x32x1. Output = 28x28x6.
    conv1_W = tf.Variable(tf.truncated_normal(shape=(5, 5, 1, 6), mean = mu, stddev = sigma))
    conv1_b = tf.Variable(tf.zeros(6))
    conv1   = tf.nn.conv2d(x, conv1_W, strides=[1, 1, 1, 1], padding='VALID') + conv1_b

    # SOLUTION: Activation.
    conv1 = tf.nn.relu(conv1)

    # SOLUTION: Pooling. Input = 28x28x6. Output = 14x14x6.
    conv1 = tf.nn.max_pool(conv1, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='VALID')

    # SOLUTION: Layer 2: Convolutional. Output = 10x10x16.
    conv2_W = tf.Variable(tf.truncated_normal(shape=(5, 5, 6, 16), mean = mu, stddev = sigma))
    conv2_b = tf.Variable(tf.zeros(16))
    conv2   = tf.nn.conv2d(conv1, conv2_W, strides=[1, 1, 1, 1], padding='VALID') + conv2_b
    
    # SOLUTION: Activation.
    conv2 = tf.nn.relu(conv2)

    # SOLUTION: Pooling. Input = 10x10x16. Output = 5x5x16.
    conv2 = tf.nn.max_pool(conv2, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='VALID')

    # SOLUTION: Flatten. Input = 5x5x16. Output = 400.
    fc0   = flatten(conv2)
    
    # SOLUTION: Layer 3: Fully Connected. Input = 400. Output = 120.
    fc1_W = tf.Variable(tf.truncated_normal(shape=(400, 120), mean = mu, stddev = sigma))
    fc1_b = tf.Variable(tf.zeros(120))
    fc1   = tf.matmul(fc0, fc1_W) + fc1_b
    
    # SOLUTION: Activation.
    fc1    = tf.nn.relu(fc1)
    
    fc1 = tf.nn.dropout(fc1, keep_prob)

    # SOLUTION: Layer 4: Fully Connected. Input = 120. Output = 84.
    fc2_W  = tf.Variable(tf.truncated_normal(shape=(120, 84), mean = mu, stddev = sigma))
    fc2_b  = tf.Variable(tf.zeros(84))
    fc2    = tf.matmul(fc1, fc2_W) + fc2_b
    
    # SOLUTION: Activation.
    fc2    = tf.nn.relu(fc2)

    fc2 = tf.nn.dropout(fc2, keep_prob)
    # SOLUTION: Layer 5: Fully Connected. Input = 84. Output = 43.
    fc3_W  = tf.Variable(tf.truncated_normal(shape=(84, 43), mean = mu, stddev = sigma))
    fc3_b  = tf.Variable(tf.zeros(43))
    logits = tf.matmul(fc2, fc3_W) + fc3_b
    
    return logits

Features and Lables

Train LeNet to classify Pickle data.

x is a placeholder for a batch of input images. y is a placeholder for a batch of output labels.

In [95]:
x = tf.placeholder(tf.float32, (None, 32, 32, 1))
y = tf.placeholder(tf.int32, (None))
one_hot_y = tf.one_hot(y, 43)
keep_prob = tf.placeholder(tf.float32)

Train, Validate and Test the Model

A validation set can be used to assess how well the model is performing. A low accuracy on the training and validation sets imply underfitting. A high accuracy on the training set but low accuracy on the validation set implies overfitting.

In [162]:
### Train your model here.
### Calculate and report the accuracy on the training and validation set.
### Once a final model architecture is selected, 
### the accuracy on the test set should be calculated and reported as well.
### Feel free to use as many code cells as needed.


rate = 0.0007
beta = 0.01

logits = LeNet(x)
cross_entropy = tf.nn.softmax_cross_entropy_with_logits(labels=one_hot_y, logits=logits)
loss_operation2 = tf.reduce_mean(cross_entropy)

regularizer = tf.nn.l2_loss(cross_entropy)
loss_operation = tf.reduce_mean(loss_operation2 + beta * regularizer)

    

optimizer = tf.train.AdamOptimizer(learning_rate = rate)
training_operation = optimizer.minimize(loss_operation)
In [163]:
correct_prediction = tf.equal(tf.argmax(logits, 1), tf.argmax(one_hot_y, 1))
accuracy_operation = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
saver = tf.train.Saver()

def evaluate(X_data, y_data):
    num_examples = len(X_data)
    total_accuracy = 0
    sess = tf.get_default_session()
    for offset in range(0, num_examples, BATCH_SIZE):
        batch_x, batch_y = X_data[offset:offset+BATCH_SIZE], y_data[offset:offset+BATCH_SIZE]
        accuracy = sess.run(accuracy_operation, feed_dict={x: batch_x, y: batch_y,keep_prob:1.0})
        total_accuracy += (accuracy * len(batch_x))
    return total_accuracy / num_examples

def evaluate_loss2(X_data, y_data):
    num_examples = len(X_data)
    total_loss = 0
    sess = tf.get_default_session()
    for offset in range(0, num_examples, BATCH_SIZE):
        batch_x, batch_y = X_data[offset:offset+BATCH_SIZE], y_data[offset:offset+BATCH_SIZE]
        loss = sess.run(loss_operation, feed_dict={x: batch_x, y: batch_y, keep_prob: 1.0})
        total_loss += (loss * len(batch_x))
    return total_loss / num_examples

def evaluate_loss(X_data, y_data):
    num_examples = len(X_data)
    total_loss = 0
    sess = tf.get_default_session()
    for offset in range(0, num_examples, BATCH_SIZE):
        batch_x, batch_y = X_data[offset:offset+BATCH_SIZE], y_data[offset:offset+BATCH_SIZE]
        loss = sess.run(loss_operation2, feed_dict={x: batch_x, y: batch_y, keep_prob: 1.0})
        total_loss += (loss * len(batch_x))
    return total_loss / num_examples
In [164]:
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    num_examples = len(X_train)
    
    print("Training...")
    print()
    validation_accuracy_list = []
    training_accuracy_list = []
    validation_loss_list = []
    training_loss_list = []
    validation_loss2_list = []
    training_loss2_list = []
    
    for i in range(EPOCHS):
        X_train, y_train = shuffle(X_train, y_train)
        for offset in range(0, num_examples, BATCH_SIZE):
            end = offset + BATCH_SIZE
            batch_x, batch_y = X_train[offset:end], y_train[offset:end]
            sess.run(training_operation, feed_dict={x: batch_x, y: batch_y,keep_prob:0.5})
            
        validation_accuracy = evaluate(X_valid, y_valid)
        train_accuracy = evaluate(X_train, y_train)
        validation_loss = evaluate_loss(X_valid, y_valid)
        train_loss = evaluate_loss(X_train, y_train)
        validation_loss2 = evaluate_loss2(X_valid, y_valid)
        train_loss2 = evaluate_loss2(X_train, y_train)
        validation_accuracy_list.append(validation_accuracy)
        training_accuracy_list.append(train_accuracy)
        validation_loss_list.append(validation_loss)
        training_loss_list.append(train_loss)
        validation_loss2_list.append(validation_loss2)
        training_loss2_list.append(train_loss2)
        
        print("EPOCH {} ...".format(i+1))
        print("Training Accuracy = {:.3f}".format(train_accuracy))
        print("Validation Accuracy = {:.3f}".format(validation_accuracy))
        print("Training Loss = {:.3f}".format(train_loss))
        print("Validation Loss = {:.3f}".format(validation_loss))
        print("Training l2 Loss = {:.3f}".format(train_loss2))
        print("Validation l2 Loss = {:.3f}".format(validation_loss2))
        print()
        
    saver.save(sess, './lenet')
    print("Model saved")
Training...

EPOCH 1 ...
Training Accuracy = 0.601
Validation Accuracy = 0.563
Training Loss = 1.616
Validation Loss = 1.712
Training l2 Loss = 3.889
Validation l2 Loss = 4.232

EPOCH 2 ...
Training Accuracy = 0.808
Validation Accuracy = 0.762
Training Loss = 0.872
Validation Loss = 0.965
Training l2 Loss = 1.738
Validation l2 Loss = 2.043

EPOCH 3 ...
Training Accuracy = 0.872
Validation Accuracy = 0.824
Training Loss = 0.609
Validation Loss = 0.703
Training l2 Loss = 1.121
Validation l2 Loss = 1.409

EPOCH 4 ...
Training Accuracy = 0.909
Validation Accuracy = 0.863
Training Loss = 0.479
Validation Loss = 0.570
Training l2 Loss = 0.837
Validation l2 Loss = 1.120

EPOCH 5 ...
Training Accuracy = 0.924
Validation Accuracy = 0.892
Training Loss = 0.387
Validation Loss = 0.476
Training l2 Loss = 0.656
Validation l2 Loss = 0.943

EPOCH 6 ...
Training Accuracy = 0.931
Validation Accuracy = 0.891
Training Loss = 0.323
Validation Loss = 0.430
Training l2 Loss = 0.539
Validation l2 Loss = 0.891

EPOCH 7 ...
Training Accuracy = 0.949
Validation Accuracy = 0.915
Training Loss = 0.287
Validation Loss = 0.389
Training l2 Loss = 0.459
Validation l2 Loss = 0.749

EPOCH 8 ...
Training Accuracy = 0.951
Validation Accuracy = 0.915
Training Loss = 0.249
Validation Loss = 0.343
Training l2 Loss = 0.399
Validation l2 Loss = 0.685

EPOCH 9 ...
Training Accuracy = 0.957
Validation Accuracy = 0.924
Training Loss = 0.223
Validation Loss = 0.317
Training l2 Loss = 0.350
Validation l2 Loss = 0.632

EPOCH 10 ...
Training Accuracy = 0.967
Validation Accuracy = 0.934
Training Loss = 0.198
Validation Loss = 0.290
Training l2 Loss = 0.309
Validation l2 Loss = 0.593

EPOCH 11 ...
Training Accuracy = 0.965
Validation Accuracy = 0.932
Training Loss = 0.199
Validation Loss = 0.288
Training l2 Loss = 0.307
Validation l2 Loss = 0.557

EPOCH 12 ...
Training Accuracy = 0.973
Validation Accuracy = 0.939
Training Loss = 0.159
Validation Loss = 0.261
Training l2 Loss = 0.243
Validation l2 Loss = 0.557

EPOCH 13 ...
Training Accuracy = 0.976
Validation Accuracy = 0.946
Training Loss = 0.152
Validation Loss = 0.248
Training l2 Loss = 0.229
Validation l2 Loss = 0.525

EPOCH 14 ...
Training Accuracy = 0.977
Validation Accuracy = 0.945
Training Loss = 0.125
Validation Loss = 0.233
Training l2 Loss = 0.192
Validation l2 Loss = 0.556

EPOCH 15 ...
Training Accuracy = 0.981
Validation Accuracy = 0.949
Training Loss = 0.132
Validation Loss = 0.224
Training l2 Loss = 0.197
Validation l2 Loss = 0.503

EPOCH 16 ...
Training Accuracy = 0.982
Validation Accuracy = 0.950
Training Loss = 0.120
Validation Loss = 0.223
Training l2 Loss = 0.177
Validation l2 Loss = 0.498

EPOCH 17 ...
Training Accuracy = 0.985
Validation Accuracy = 0.955
Training Loss = 0.107
Validation Loss = 0.208
Training l2 Loss = 0.156
Validation l2 Loss = 0.485

EPOCH 18 ...
Training Accuracy = 0.987
Validation Accuracy = 0.954
Training Loss = 0.095
Validation Loss = 0.196
Training l2 Loss = 0.138
Validation l2 Loss = 0.484

EPOCH 19 ...
Training Accuracy = 0.987
Validation Accuracy = 0.950
Training Loss = 0.088
Validation Loss = 0.203
Training l2 Loss = 0.130
Validation l2 Loss = 0.520

EPOCH 20 ...
Training Accuracy = 0.987
Validation Accuracy = 0.955
Training Loss = 0.087
Validation Loss = 0.188
Training l2 Loss = 0.126
Validation l2 Loss = 0.451

EPOCH 21 ...
Training Accuracy = 0.989
Validation Accuracy = 0.952
Training Loss = 0.077
Validation Loss = 0.187
Training l2 Loss = 0.110
Validation l2 Loss = 0.479

EPOCH 22 ...
Training Accuracy = 0.989
Validation Accuracy = 0.950
Training Loss = 0.073
Validation Loss = 0.190
Training l2 Loss = 0.105
Validation l2 Loss = 0.517

EPOCH 23 ...
Training Accuracy = 0.991
Validation Accuracy = 0.961
Training Loss = 0.061
Validation Loss = 0.167
Training l2 Loss = 0.087
Validation l2 Loss = 0.457

EPOCH 24 ...
Training Accuracy = 0.990
Validation Accuracy = 0.957
Training Loss = 0.071
Validation Loss = 0.177
Training l2 Loss = 0.099
Validation l2 Loss = 0.452

EPOCH 25 ...
Training Accuracy = 0.992
Validation Accuracy = 0.958
Training Loss = 0.059
Validation Loss = 0.169
Training l2 Loss = 0.082
Validation l2 Loss = 0.469

EPOCH 26 ...
Training Accuracy = 0.992
Validation Accuracy = 0.959
Training Loss = 0.059
Validation Loss = 0.167
Training l2 Loss = 0.082
Validation l2 Loss = 0.454

EPOCH 27 ...
Training Accuracy = 0.992
Validation Accuracy = 0.960
Training Loss = 0.056
Validation Loss = 0.164
Training l2 Loss = 0.078
Validation l2 Loss = 0.457

EPOCH 28 ...
Training Accuracy = 0.992
Validation Accuracy = 0.960
Training Loss = 0.049
Validation Loss = 0.160
Training l2 Loss = 0.069
Validation l2 Loss = 0.478

EPOCH 29 ...
Training Accuracy = 0.992
Validation Accuracy = 0.956
Training Loss = 0.048
Validation Loss = 0.168
Training l2 Loss = 0.068
Validation l2 Loss = 0.507

EPOCH 30 ...
Training Accuracy = 0.994
Validation Accuracy = 0.967
Training Loss = 0.040
Validation Loss = 0.137
Training l2 Loss = 0.056
Validation l2 Loss = 0.412

EPOCH 31 ...
Training Accuracy = 0.994
Validation Accuracy = 0.958
Training Loss = 0.038
Validation Loss = 0.160
Training l2 Loss = 0.054
Validation l2 Loss = 0.508

EPOCH 32 ...
Training Accuracy = 0.995
Validation Accuracy = 0.963
Training Loss = 0.037
Validation Loss = 0.141
Training l2 Loss = 0.051
Validation l2 Loss = 0.419

EPOCH 33 ...
Training Accuracy = 0.996
Validation Accuracy = 0.963
Training Loss = 0.034
Validation Loss = 0.146
Training l2 Loss = 0.046
Validation l2 Loss = 0.470

EPOCH 34 ...
Training Accuracy = 0.995
Validation Accuracy = 0.964
Training Loss = 0.031
Validation Loss = 0.144
Training l2 Loss = 0.042
Validation l2 Loss = 0.507

EPOCH 35 ...
Training Accuracy = 0.995
Validation Accuracy = 0.957
Training Loss = 0.035
Validation Loss = 0.158
Training l2 Loss = 0.048
Validation l2 Loss = 0.543

EPOCH 36 ...
Training Accuracy = 0.996
Validation Accuracy = 0.965
Training Loss = 0.033
Validation Loss = 0.142
Training l2 Loss = 0.044
Validation l2 Loss = 0.506

EPOCH 37 ...
Training Accuracy = 0.996
Validation Accuracy = 0.961
Training Loss = 0.028
Validation Loss = 0.144
Training l2 Loss = 0.039
Validation l2 Loss = 0.520

EPOCH 38 ...
Training Accuracy = 0.997
Validation Accuracy = 0.970
Training Loss = 0.027
Validation Loss = 0.123
Training l2 Loss = 0.036
Validation l2 Loss = 0.397

EPOCH 39 ...
Training Accuracy = 0.996
Validation Accuracy = 0.969
Training Loss = 0.025
Validation Loss = 0.135
Training l2 Loss = 0.034
Validation l2 Loss = 0.509

EPOCH 40 ...
Training Accuracy = 0.997
Validation Accuracy = 0.965
Training Loss = 0.024
Validation Loss = 0.143
Training l2 Loss = 0.032
Validation l2 Loss = 0.546

EPOCH 41 ...
Training Accuracy = 0.997
Validation Accuracy = 0.969
Training Loss = 0.022
Validation Loss = 0.128
Training l2 Loss = 0.029
Validation l2 Loss = 0.475

EPOCH 42 ...
Training Accuracy = 0.998
Validation Accuracy = 0.970
Training Loss = 0.023
Validation Loss = 0.118
Training l2 Loss = 0.030
Validation l2 Loss = 0.371

EPOCH 43 ...
Training Accuracy = 0.997
Validation Accuracy = 0.966
Training Loss = 0.023
Validation Loss = 0.128
Training l2 Loss = 0.029
Validation l2 Loss = 0.446

EPOCH 44 ...
Training Accuracy = 0.997
Validation Accuracy = 0.968
Training Loss = 0.021
Validation Loss = 0.130
Training l2 Loss = 0.027
Validation l2 Loss = 0.494

EPOCH 45 ...
Training Accuracy = 0.998
Validation Accuracy = 0.965
Training Loss = 0.017
Validation Loss = 0.145
Training l2 Loss = 0.022
Validation l2 Loss = 0.593

EPOCH 46 ...
Training Accuracy = 0.997
Validation Accuracy = 0.965
Training Loss = 0.019
Validation Loss = 0.156
Training l2 Loss = 0.025
Validation l2 Loss = 0.695

EPOCH 47 ...
Training Accuracy = 0.998
Validation Accuracy = 0.968
Training Loss = 0.018
Validation Loss = 0.141
Training l2 Loss = 0.023
Validation l2 Loss = 0.606

EPOCH 48 ...
Training Accuracy = 0.997
Validation Accuracy = 0.971
Training Loss = 0.019
Validation Loss = 0.126
Training l2 Loss = 0.025
Validation l2 Loss = 0.495

EPOCH 49 ...
Training Accuracy = 0.998
Validation Accuracy = 0.971
Training Loss = 0.017
Validation Loss = 0.138
Training l2 Loss = 0.022
Validation l2 Loss = 0.616

EPOCH 50 ...
Training Accuracy = 0.998
Validation Accuracy = 0.968
Training Loss = 0.017
Validation Loss = 0.138
Training l2 Loss = 0.021
Validation l2 Loss = 0.596

EPOCH 51 ...
Training Accuracy = 0.998
Validation Accuracy = 0.965
Training Loss = 0.017
Validation Loss = 0.128
Training l2 Loss = 0.021
Validation l2 Loss = 0.465

EPOCH 52 ...
Training Accuracy = 0.998
Validation Accuracy = 0.967
Training Loss = 0.014
Validation Loss = 0.136
Training l2 Loss = 0.018
Validation l2 Loss = 0.599

EPOCH 53 ...
Training Accuracy = 0.998
Validation Accuracy = 0.970
Training Loss = 0.015
Validation Loss = 0.129
Training l2 Loss = 0.020
Validation l2 Loss = 0.535

EPOCH 54 ...
Training Accuracy = 0.999
Validation Accuracy = 0.969
Training Loss = 0.012
Validation Loss = 0.143
Training l2 Loss = 0.015
Validation l2 Loss = 0.690

EPOCH 55 ...
Training Accuracy = 0.998
Validation Accuracy = 0.974
Training Loss = 0.014
Validation Loss = 0.113
Training l2 Loss = 0.019
Validation l2 Loss = 0.473

EPOCH 56 ...
Training Accuracy = 0.999
Validation Accuracy = 0.969
Training Loss = 0.013
Validation Loss = 0.132
Training l2 Loss = 0.017
Validation l2 Loss = 0.563

EPOCH 57 ...
Training Accuracy = 0.999
Validation Accuracy = 0.974
Training Loss = 0.014
Validation Loss = 0.118
Training l2 Loss = 0.017
Validation l2 Loss = 0.516

EPOCH 58 ...
Training Accuracy = 0.999
Validation Accuracy = 0.975
Training Loss = 0.011
Validation Loss = 0.115
Training l2 Loss = 0.014
Validation l2 Loss = 0.513

EPOCH 59 ...
Training Accuracy = 0.998
Validation Accuracy = 0.962
Training Loss = 0.016
Validation Loss = 0.147
Training l2 Loss = 0.021
Validation l2 Loss = 0.601

EPOCH 60 ...
Training Accuracy = 0.999
Validation Accuracy = 0.969
Training Loss = 0.012
Validation Loss = 0.130
Training l2 Loss = 0.015
Validation l2 Loss = 0.594

EPOCH 61 ...
Training Accuracy = 0.999
Validation Accuracy = 0.968
Training Loss = 0.010
Validation Loss = 0.137
Training l2 Loss = 0.012
Validation l2 Loss = 0.673

EPOCH 62 ...
Training Accuracy = 0.999
Validation Accuracy = 0.966
Training Loss = 0.010
Validation Loss = 0.138
Training l2 Loss = 0.012
Validation l2 Loss = 0.681

EPOCH 63 ...
Training Accuracy = 0.999
Validation Accuracy = 0.970
Training Loss = 0.010
Validation Loss = 0.126
Training l2 Loss = 0.013
Validation l2 Loss = 0.613

EPOCH 64 ...
Training Accuracy = 0.999
Validation Accuracy = 0.970
Training Loss = 0.010
Validation Loss = 0.134
Training l2 Loss = 0.012
Validation l2 Loss = 0.647

EPOCH 65 ...
Training Accuracy = 0.999
Validation Accuracy = 0.973
Training Loss = 0.012
Validation Loss = 0.118
Training l2 Loss = 0.015
Validation l2 Loss = 0.489

EPOCH 66 ...
Training Accuracy = 0.999
Validation Accuracy = 0.970
Training Loss = 0.010
Validation Loss = 0.130
Training l2 Loss = 0.012
Validation l2 Loss = 0.600

EPOCH 67 ...
Training Accuracy = 0.999
Validation Accuracy = 0.968
Training Loss = 0.010
Validation Loss = 0.147
Training l2 Loss = 0.012
Validation l2 Loss = 0.774

EPOCH 68 ...
Training Accuracy = 0.999
Validation Accuracy = 0.968
Training Loss = 0.008
Validation Loss = 0.140
Training l2 Loss = 0.010
Validation l2 Loss = 0.771

EPOCH 69 ...
Training Accuracy = 0.999
Validation Accuracy = 0.971
Training Loss = 0.011
Validation Loss = 0.151
Training l2 Loss = 0.015
Validation l2 Loss = 0.935

EPOCH 70 ...
Training Accuracy = 0.999
Validation Accuracy = 0.971
Training Loss = 0.008
Validation Loss = 0.138
Training l2 Loss = 0.009
Validation l2 Loss = 0.795

EPOCH 71 ...
Training Accuracy = 0.998
Validation Accuracy = 0.971
Training Loss = 0.012
Validation Loss = 0.139
Training l2 Loss = 0.016
Validation l2 Loss = 0.739

EPOCH 72 ...
Training Accuracy = 0.999
Validation Accuracy = 0.967
Training Loss = 0.009
Validation Loss = 0.149
Training l2 Loss = 0.012
Validation l2 Loss = 0.822

EPOCH 73 ...
Training Accuracy = 0.998
Validation Accuracy = 0.961
Training Loss = 0.010
Validation Loss = 0.172
Training l2 Loss = 0.014
Validation l2 Loss = 0.866

EPOCH 74 ...
Training Accuracy = 0.999
Validation Accuracy = 0.973
Training Loss = 0.008
Validation Loss = 0.123
Training l2 Loss = 0.011
Validation l2 Loss = 0.647

EPOCH 75 ...
Training Accuracy = 0.999
Validation Accuracy = 0.971
Training Loss = 0.008
Validation Loss = 0.147
Training l2 Loss = 0.009
Validation l2 Loss = 0.846

EPOCH 76 ...
Training Accuracy = 1.000
Validation Accuracy = 0.973
Training Loss = 0.006
Validation Loss = 0.136
Training l2 Loss = 0.008
Validation l2 Loss = 0.865

EPOCH 77 ...
Training Accuracy = 1.000
Validation Accuracy = 0.972
Training Loss = 0.006
Validation Loss = 0.140
Training l2 Loss = 0.007
Validation l2 Loss = 0.852

EPOCH 78 ...
Training Accuracy = 0.999
Validation Accuracy = 0.973
Training Loss = 0.007
Validation Loss = 0.133
Training l2 Loss = 0.008
Validation l2 Loss = 0.771

EPOCH 79 ...
Training Accuracy = 0.999
Validation Accuracy = 0.973
Training Loss = 0.007
Validation Loss = 0.135
Training l2 Loss = 0.008
Validation l2 Loss = 0.795

EPOCH 80 ...
Training Accuracy = 0.999
Validation Accuracy = 0.976
Training Loss = 0.007
Validation Loss = 0.124
Training l2 Loss = 0.009
Validation l2 Loss = 0.762

EPOCH 81 ...
Training Accuracy = 1.000
Validation Accuracy = 0.973
Training Loss = 0.007
Validation Loss = 0.137
Training l2 Loss = 0.009
Validation l2 Loss = 0.829

EPOCH 82 ...
Training Accuracy = 1.000
Validation Accuracy = 0.974
Training Loss = 0.006
Validation Loss = 0.141
Training l2 Loss = 0.007
Validation l2 Loss = 0.880

EPOCH 83 ...
Training Accuracy = 0.999
Validation Accuracy = 0.970
Training Loss = 0.006
Validation Loss = 0.140
Training l2 Loss = 0.008
Validation l2 Loss = 0.831

EPOCH 84 ...
Training Accuracy = 1.000
Validation Accuracy = 0.971
Training Loss = 0.007
Validation Loss = 0.139
Training l2 Loss = 0.009
Validation l2 Loss = 0.823

EPOCH 85 ...
Training Accuracy = 1.000
Validation Accuracy = 0.975
Training Loss = 0.005
Validation Loss = 0.126
Training l2 Loss = 0.006
Validation l2 Loss = 0.787

EPOCH 86 ...
Training Accuracy = 1.000
Validation Accuracy = 0.976
Training Loss = 0.006
Validation Loss = 0.123
Training l2 Loss = 0.007
Validation l2 Loss = 0.732

EPOCH 87 ...
Training Accuracy = 0.999
Validation Accuracy = 0.973
Training Loss = 0.007
Validation Loss = 0.132
Training l2 Loss = 0.009
Validation l2 Loss = 0.799

EPOCH 88 ...
Training Accuracy = 0.999
Validation Accuracy = 0.976
Training Loss = 0.005
Validation Loss = 0.137
Training l2 Loss = 0.006
Validation l2 Loss = 0.904

EPOCH 89 ...
Training Accuracy = 0.999
Validation Accuracy = 0.973
Training Loss = 0.007
Validation Loss = 0.125
Training l2 Loss = 0.008
Validation l2 Loss = 0.712

EPOCH 90 ...
Training Accuracy = 1.000
Validation Accuracy = 0.976
Training Loss = 0.005
Validation Loss = 0.120
Training l2 Loss = 0.006
Validation l2 Loss = 0.745

EPOCH 91 ...
Training Accuracy = 0.999
Validation Accuracy = 0.978
Training Loss = 0.008
Validation Loss = 0.116
Training l2 Loss = 0.011
Validation l2 Loss = 0.725

EPOCH 92 ...
Training Accuracy = 1.000
Validation Accuracy = 0.971
Training Loss = 0.005
Validation Loss = 0.132
Training l2 Loss = 0.006
Validation l2 Loss = 0.778

EPOCH 93 ...
Training Accuracy = 1.000
Validation Accuracy = 0.976
Training Loss = 0.005
Validation Loss = 0.121
Training l2 Loss = 0.006
Validation l2 Loss = 0.746

EPOCH 94 ...
Training Accuracy = 1.000
Validation Accuracy = 0.973
Training Loss = 0.004
Validation Loss = 0.135
Training l2 Loss = 0.005
Validation l2 Loss = 0.874

EPOCH 95 ...
Training Accuracy = 1.000
Validation Accuracy = 0.975
Training Loss = 0.005
Validation Loss = 0.119
Training l2 Loss = 0.005
Validation l2 Loss = 0.693

EPOCH 96 ...
Training Accuracy = 1.000
Validation Accuracy = 0.974
Training Loss = 0.006
Validation Loss = 0.138
Training l2 Loss = 0.007
Validation l2 Loss = 0.807

EPOCH 97 ...
Training Accuracy = 1.000
Validation Accuracy = 0.974
Training Loss = 0.006
Validation Loss = 0.127
Training l2 Loss = 0.007
Validation l2 Loss = 0.819

EPOCH 98 ...
Training Accuracy = 1.000
Validation Accuracy = 0.976
Training Loss = 0.004
Validation Loss = 0.125
Training l2 Loss = 0.005
Validation l2 Loss = 0.832

EPOCH 99 ...
Training Accuracy = 1.000
Validation Accuracy = 0.977
Training Loss = 0.004
Validation Loss = 0.131
Training l2 Loss = 0.004
Validation l2 Loss = 0.886

EPOCH 100 ...
Training Accuracy = 1.000
Validation Accuracy = 0.976
Training Loss = 0.004
Validation Loss = 0.128
Training l2 Loss = 0.005
Validation l2 Loss = 0.857

Model saved
In [166]:
import os
import stat
target_accuracy = [.93] * EPOCHS

    
plt.plot(validation_accuracy_list, label='Validation')
plt.plot(training_accuracy_list, label='Training')
target_line, = plt.plot(target_accuracy, '-', label='Target', color='r')
target_line.set_dashes([1, 1])
plt.xlabel('Epoch')
plt.ylabel('Accuracy')
plt.grid(True)
plt.legend()
filename = 'examples/grey_norm_filter_drop_training.png'

plt.savefig(filename)
os.chmod(filename, stat.S_IRUSR | stat.S_IRGRP | stat.S_IROTH)
plt.show()
In [167]:
import os
import stat
#target_accuracy = [.93] * EPOCHS

    
plt.plot(validation_loss_list, label='Validation Loss')
plt.plot(training_loss_list, label='Training Loss')
plt.plot(validation_loss2_list, label='Validation L2 Loss')
plt.plot(training_loss2_list, label='Training L2 Loss')

plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.grid(True)
plt.legend()
filename = 'examples/grey_norm_filter_drop_training_loss.png'

plt.savefig(filename)
os.chmod(filename, stat.S_IRUSR | stat.S_IRGRP | stat.S_IROTH)
plt.show()
In [168]:
with tf.Session() as sess:
    saver.restore(sess, tf.train.latest_checkpoint('.'))

    train_accuracy = evaluate(X_train, y_train)
    print("Train Accuracy = {:.3f}".format(train_accuracy))
    
    valid_accuracy = evaluate(X_valid, y_valid)
    print("Valid Accuracy = {:.3f}".format(valid_accuracy))    
    
    test_accuracy = evaluate(X_test, y_test)
    print("Test Accuracy = {:.3f}".format(test_accuracy))
Train Accuracy = 1.000
Valid Accuracy = 0.976
Test Accuracy = 0.950

Step 3: Test a Model on New Images

To give yourself more insight into how your model is working, download at least five pictures of German traffic signs from the web and use your model to predict the traffic sign type.

You may find signnames.csv useful as it contains mappings from the class id (integer) to the actual sign name.

Load and Output the Images

In [324]:
### Load the images and plot them here.
### Feel free to use as many code cells as needed.
import PIL
from PIL import Image
import glob


def PIL2array(img):
    return np.array(img.getdata(),np.uint8).reshape(img.size[1], img.size[0], 3)


def open_images(path):
    german_images = []
    for image in (glob.glob(path)):
        german_images.append(Image.open(image))
    return german_images

def display_images(image_list):
    plt.figure(figsize=(35,35))
    for i, image in enumerate(image_list):
        ax = plt.subplot(1, len(image_list), i+1)
        ax.axis('off')
        ax.imshow(image)
    plt.show()

def display_grey_images(image_list):
    plt.figure(figsize=(35,35))
    for i, image in enumerate(image_list):
        ax = plt.subplot(1, len(image_list), i+1)
        ax.axis('off')
        ax.imshow(image.squeeze(), cmap = 'gray')
        
    plt.show()
    
def resize_image(image):
    return image.resize((32,32), Image.ANTIALIAS)
def process_images(image_list):
    processed_image_list = []
    for image in image_list:
        processed_image = resize_image(image)
        processed_image = PIL2array(processed_image)
        processed_image = greyscale_single(processed_image)
        processed_image = histogram_equalization(processed_image, False)
        processed_image = normalize(processed_image)
        processed_image_list.append(processed_image)
    return processed_image_list


    

raw_image_list = open_images('./german_signs/*.jpg')

print("Ram Images")
display_images(raw_image_list)
print("Processed Images")
processed_image_list = process_images(raw_image_list)
display_grey_images(processed_image_list)
processed_image_list_labels = [13, 2, 25, 17, 1, 3, 9, 9]
Ram Images
Processed Images

Predict the Sign Type for Each Image

In [325]:
### Run the predictions here and use the model to output the prediction for each image.
### Make sure to pre-process the images with the same pre-processing pipeline used earlier.
### Feel free to use as many code cells as needed.

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    saver.restore(sess, "./lenet")
    accuracy = evaluate(processed_image_list, processed_image_list_labels)
    print("Data Set Accuracy = {:.3f}".format(accuracy))
Data Set Accuracy = 0.625

Analyze Performance

In [326]:
### Calculate the accuracy for these 5 new images. 
### For example, if the model predicted 1 out of 5 signs correctly, it's 20% accurate on these new images.

my_single_item_array = []
my_single_item_label_array = []

for i, img in enumerate(processed_image_list):
    my_single_item_array.append(processed_image_list[i])
    my_single_item_label_array.append(processed_image_list_labels[i])

    with tf.Session() as sess:
        sess.run(tf.global_variables_initializer())
        saver.restore(sess, "./lenet")
        my_accuracy = evaluate(my_single_item_array, my_single_item_label_array)
        print('Image {}'.format(i+1))
        print("Image Accuracy = {:.3f}".format(my_accuracy))
        print()
Image 1
Image Accuracy = 1.000

Image 2
Image Accuracy = 1.000

Image 3
Image Accuracy = 1.000

Image 4
Image Accuracy = 1.000

Image 5
Image Accuracy = 1.000

Image 6
Image Accuracy = 0.833

Image 7
Image Accuracy = 0.714

Image 8
Image Accuracy = 0.625

Output Top 5 Softmax Probabilities For Each Image Found on the Web

For each of the new images, print out the model's softmax probabilities to show the certainty of the model's predictions (limit the output to the top 5 probabilities for each image). tf.nn.top_k could prove helpful here.

The example below demonstrates how tf.nn.top_k can be used to find the top k predictions for each image.

tf.nn.top_k will return the values and indices (class ids) of the top k predictions. So if k=3, for each sign, it'll return the 3 largest probabilities (out of a possible 43) and the correspoding class ids.

Take this numpy array as an example. The values in the array represent predictions. The array contains softmax probabilities for five candidate images with six possible classes. tk.nn.top_k is used to choose the three classes with the highest probability:

# (5, 6) array
a = np.array([[ 0.24879643,  0.07032244,  0.12641572,  0.34763842,  0.07893497,
         0.12789202],
       [ 0.28086119,  0.27569815,  0.08594638,  0.0178669 ,  0.18063401,
         0.15899337],
       [ 0.26076848,  0.23664738,  0.08020603,  0.07001922,  0.1134371 ,
         0.23892179],
       [ 0.11943333,  0.29198961,  0.02605103,  0.26234032,  0.1351348 ,
         0.16505091],
       [ 0.09561176,  0.34396535,  0.0643941 ,  0.16240774,  0.24206137,
         0.09155967]])

Running it through sess.run(tf.nn.top_k(tf.constant(a), k=3)) produces:

TopKV2(values=array([[ 0.34763842,  0.24879643,  0.12789202],
       [ 0.28086119,  0.27569815,  0.18063401],
       [ 0.26076848,  0.23892179,  0.23664738],
       [ 0.29198961,  0.26234032,  0.16505091],
       [ 0.34396535,  0.24206137,  0.16240774]]), indices=array([[3, 0, 5],
       [0, 1, 4],
       [0, 5, 1],
       [1, 3, 5],
       [1, 4, 3]], dtype=int32))

Looking just at the first row we get [ 0.34763842, 0.24879643, 0.12789202], you can confirm these are the 3 largest probabilities in a. You'll also notice [3, 0, 5] are the corresponding indices.

In [327]:
### Print out the top five softmax probabilities for the predictions on the German traffic sign images found on the web. 
### Feel free to use as many code cells as needed.

def display_images_guess(image_list, labels):
    plt.figure(figsize=(35,35))
    for i, image in enumerate(image_list):
        ax = plt.subplot(6, len(image_list), i+1)
        ax.axis('off')
        ax.set_title(labels[i], fontsize=20)
        ax.imshow(image)
    plt.show()
    
k_size = 5
softmax_logits = tf.nn.softmax(logits)
top_k = tf.nn.top_k(softmax_logits, k=k_size)

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    saver.restore(sess, "./lenet")
    my_softmax_logits = sess.run(softmax_logits, feed_dict={x: processed_image_list, keep_prob: 1.0})
    my_top_k = sess.run(top_k, feed_dict={x: processed_image_list, keep_prob: 1.0})

    for i in range(len(processed_image_list)):
        figures = []
        labels = []
        
        figures.append(raw_image_list[i])
        labels.append("Original")
        
        for j in range(k_size):
            labels.append('Guess {} : ({:.0f}%)'.format(j+1, 100*my_top_k[0][i][j]))
            figures.append(X_valid[np.argwhere(y_valid == my_top_k[1][i][j])[0]].squeeze())
            
        display_images_guess(figures, labels)

Project Writeup

Once you have completed the code implementation, document your results in a project writeup using this template as a guide. The writeup can be in a markdown or pdf file.

Note: Once you have completed all of the code implementations and successfully answered each question above, you may finalize your work by exporting the iPython Notebook as an HTML document. You can do this by using the menu above and navigating to \n", "File -> Download as -> HTML (.html). Include the finished document along with this notebook as your submission.


Step 4 (Optional): Visualize the Neural Network's State with Test Images

This Section is not required to complete but acts as an additional excersise for understaning the output of a neural network's weights. While neural networks can be a great learning device they are often referred to as a black box. We can understand what the weights of a neural network look like better by plotting their feature maps. After successfully training your neural network you can see what it's feature maps look like by plotting the output of the network's weight layers in response to a test stimuli image. From these plotted feature maps, it's possible to see what characteristics of an image the network finds interesting. For a sign, maybe the inner network feature maps react with high activation to the sign's boundary outline or to the contrast in the sign's painted symbol.

Provided for you below is the function code that allows you to get the visualization output of any tensorflow weight layer you want. The inputs to the function should be a stimuli image, one used during training or a new one you provided, and then the tensorflow variable name that represents the layer's state during the training process, for instance if you wanted to see what the LeNet lab's feature maps looked like for it's second convolutional layer you could enter conv2 as the tf_activation variable.

For an example of what feature map outputs look like, check out NVIDIA's results in their paper End-to-End Deep Learning for Self-Driving Cars in the section Visualization of internal CNN State. NVIDIA was able to show that their network's inner weights had high activations to road boundary lines by comparing feature maps from an image with a clear path to one without. Try experimenting with a similar test to show that your trained network's weights are looking for interesting features, whether it's looking at differences in feature maps from images with or without a sign, or even what feature maps look like in a trained network vs a completely untrained one on the same sign image.

Combined Image

Your output should look something like this (above)

In [328]:
### Visualize your network's feature maps here.
### Feel free to use as many code cells as needed.

# image_input: the test image being fed into the network to produce the feature maps
# tf_activation: should be a tf variable name used during your training procedure that represents the calculated state of a specific weight layer
# activation_min/max: can be used to view the activation contrast in more detail, by default matplot sets min and max to the actual min and max values of the output
# plt_num: used to plot out multiple different weight feature map sets on the same block, just extend the plt number for each new feature map entry

def outputFeatureMap(image_input, tf_activation, activation_min=-1, activation_max=-1 ,plt_num=1):
    # Here make sure to preprocess your image_input in a way your network expects
    # with size, normalization, ect if needed
    # image_input =
    # Note: x should be the same name as your network's tensorflow data placeholder variable
    # If you get an error tf_activation is not defined it may be having trouble accessing the variable from inside a function
    activation = tf_activation.eval(session=sess,feed_dict={x : image_input})
    featuremaps = activation.shape[3]
    plt.figure(plt_num, figsize=(15,15))
    for featuremap in range(featuremaps):
        plt.subplot(6,8, featuremap+1) # sets the number of feature maps to show on each row and column
        plt.title('FeatureMap ' + str(featuremap)) # displays the feature map number
        if activation_min != -1 & activation_max != -1:
            plt.imshow(activation[0,:,:, featuremap], interpolation="nearest", vmin =activation_min, vmax=activation_max, cmap="gray")
        elif activation_max != -1:
            plt.imshow(activation[0,:,:, featuremap], interpolation="nearest", vmax=activation_max, cmap="gray")
        elif activation_min !=-1:
            plt.imshow(activation[0,:,:, featuremap], interpolation="nearest", vmin=activation_min, cmap="gray")
        else:
            plt.imshow(activation[0,:,:, featuremap], interpolation="nearest", cmap="gray")
In [ ]: